{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "Congratulations for completing the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program!  In this notebook, you will learn how to control an agent in a more challenging environment, where the goal is to train a creature with four arms to walk forward.  **Note that this exercise is optional!**\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Crawler.app\"`\n",
    "- **Windows** (x86): `\"path/to/Crawler_Windows_x86/Crawler.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Crawler_Windows_x86_64/Crawler.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Crawler_Linux/Crawler.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Crawler_Linux/Crawler.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Crawler_Linux_NoVis/Crawler.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Crawler_Linux_NoVis/Crawler.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Crawler.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Crawler.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: CrawlerBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 129\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 20\n",
      "        Vector Action descriptions: , , , , , , , , , , , , , , , , , , , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name='Crawler.exe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 12\n",
      "Size of each action: 20\n",
      "There are 12 agents. Each observes a state with length: 129\n",
      "The state for the first agent looks like: [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  2.25000000e+00\n",
      "  1.00000000e+00  0.00000000e+00  1.78813934e-07  0.00000000e+00\n",
      "  1.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  6.06093168e-01 -1.42857209e-01 -6.06078804e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  1.33339906e+00 -1.42857209e-01\n",
      " -1.33341408e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      " -6.06093347e-01 -1.42857209e-01 -6.06078625e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.33339953e+00 -1.42857209e-01\n",
      " -1.33341372e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      " -6.06093168e-01 -1.42857209e-01  6.06078804e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.33339906e+00 -1.42857209e-01\n",
      "  1.33341408e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  6.06093347e-01 -1.42857209e-01  6.06078625e-01  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  1.33339953e+00 -1.42857209e-01\n",
      "  1.33341372e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agent's performance, if it selects an action at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  \n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "#states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "#scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "#while True:\n",
    "#    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "#    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "#    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "#    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "#    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "#    dones = env_info.local_done                        # see if episode finished\n",
    "#    scores += env_info.rewards                         # update the score (for each agent)\n",
    "#    states = next_states                               # roll over states to next time step\n",
    "#    if np.any(dones):                                  # exit loop if episode finished\n",
    "#        break\n",
    "#print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from  collections  import deque\n",
    "from itertools import count\n",
    "import torch\n",
    "import time\n",
    "#from ddpg_agent import Agent\n",
    "from ppo_agent import Agent\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "agent = Agent(state_size=state_size, action_size=action_size, random_seed=8, n_agent=num_agents, \\\n",
    "              fc1_units=128, fc2_units=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1, Score: 48.61, Max: 48.61, Min: 48.61, T-max: 1000   \n",
      "Episode: 2, Score: 47.50, Max: 48.61, Min: 47.50, T-max: 1000   \n",
      "Episode: 3, Score: 45.66, Max: 48.61, Min: 45.66, T-max: 1000   \n",
      "Episode: 4, Score: 48.32, Max: 48.61, Min: 45.66, T-max: 1000   \n",
      "Episode: 5, Score: 49.08, Max: 49.08, Min: 45.66, T-max: 1000   \n",
      "Episode: 6, Score: 49.45, Max: 49.45, Min: 45.66, T-max: 1000   \n",
      "Episode: 7, Score: 49.53, Max: 49.53, Min: 45.66, T-max: 1000   \n",
      "Episode: 8, Score: 51.14, Max: 51.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 9, Score: 50.75, Max: 51.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 10, Score: 46.26, Max: 51.14, Min: 45.66, T-max: 1000   \n",
      "*** Episode 10 \t Average Score (over agents): 46.26 \t Average Score on 100 Episode: 48.63, Time: 00:02:06***\n",
      "Episode: 11, Score: 47.03, Max: 51.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 12, Score: 49.65, Max: 51.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 13, Score: 53.39, Max: 53.39, Min: 45.66, T-max: 1000   \n",
      "Episode: 14, Score: 53.76, Max: 53.76, Min: 45.66, T-max: 1000   \n",
      "Episode: 15, Score: 51.71, Max: 53.76, Min: 45.66, T-max: 1000   \n",
      "Episode: 16, Score: 54.96, Max: 54.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 17, Score: 54.30, Max: 54.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 18, Score: 53.48, Max: 54.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 19, Score: 53.13, Max: 54.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 20, Score: 57.29, Max: 57.29, Min: 45.66, T-max: 1000   \n",
      "*** Episode 20 \t Average Score (over agents): 57.29 \t Average Score on 100 Episode: 50.75, Time: 00:04:17***\n",
      "Episode: 21, Score: 52.79, Max: 57.29, Min: 45.66, T-max: 1000   \n",
      "Episode: 22, Score: 52.51, Max: 57.29, Min: 45.66, T-max: 1000   \n",
      "Episode: 23, Score: 57.20, Max: 57.29, Min: 45.66, T-max: 1000   \n",
      "Episode: 24, Score: 107.98, Max: 107.98, Min: 45.66, T-max: 1999   \n",
      "Episode: 25, Score: 58.27, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 26, Score: 55.97, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 27, Score: 55.51, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 28, Score: 55.87, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 29, Score: 56.41, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 30, Score: 57.10, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 30 \t Average Score (over agents): 57.10 \t Average Score on 100 Episode: 54.15, Time: 00:06:41***\n",
      "Episode: 31, Score: 60.13, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 32, Score: 56.94, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 33, Score: 52.80, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 34, Score: 64.32, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 35, Score: 59.08, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 36, Score: 52.60, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 37, Score: 60.61, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 38, Score: 59.86, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 39, Score: 56.63, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 40, Score: 56.11, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 40 \t Average Score (over agents): 56.11 \t Average Score on 100 Episode: 55.09, Time: 00:08:57***\n",
      "Episode: 41, Score: 58.01, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 42, Score: 61.22, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 43, Score: 62.21, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 44, Score: 62.52, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 45, Score: 61.94, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 46, Score: 58.17, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 47, Score: 63.81, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 48, Score: 62.80, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 49, Score: 67.10, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 50, Score: 66.93, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 50 \t Average Score (over agents): 66.93 \t Average Score on 100 Episode: 56.57, Time: 00:11:14***\n",
      "Episode: 51, Score: 63.05, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 52, Score: 65.90, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 53, Score: 66.75, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 54, Score: 67.92, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 55, Score: 69.89, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 56, Score: 69.84, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 57, Score: 69.13, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 58, Score: 66.15, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 59, Score: 68.83, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 60, Score: 65.87, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 60 \t Average Score (over agents): 65.87 \t Average Score on 100 Episode: 58.36, Time: 00:13:31***\n",
      "Episode: 61, Score: 68.35, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 62, Score: 66.47, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 63, Score: 66.83, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 64, Score: 75.38, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 65, Score: 66.64, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 66, Score: 59.91, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 67, Score: 71.04, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 68, Score: 76.62, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 69, Score: 72.07, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 70, Score: 68.02, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 70 \t Average Score (over agents): 68.02 \t Average Score on 100 Episode: 59.90, Time: 00:15:42***\n",
      "Episode: 71, Score: 74.09, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 72, Score: 73.02, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 73, Score: 78.71, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 74, Score: 56.06, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 75, Score: 70.54, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 76, Score: 71.66, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 77, Score: 71.02, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 78, Score: 77.74, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 79, Score: 80.36, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 80, Score: 69.26, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 80 \t Average Score (over agents): 69.26 \t Average Score on 100 Episode: 61.44, Time: 00:18:02***\n",
      "Episode: 81, Score: 77.52, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 82, Score: 80.57, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 83, Score: 71.07, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 84, Score: 77.34, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 85, Score: 74.20, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 86, Score: 65.32, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 87, Score: 74.80, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 88, Score: 77.05, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 89, Score: 78.59, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 90, Score: 79.43, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 90 \t Average Score (over agents): 79.43 \t Average Score on 100 Episode: 63.02, Time: 00:20:29***\n",
      "Episode: 91, Score: 90.58, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 92, Score: 83.91, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 93, Score: 87.73, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 94, Score: 90.11, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 95, Score: 81.61, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 96, Score: 83.91, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 97, Score: 102.21, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 98, Score: 87.14, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 99, Score: 93.58, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 100, Score: 95.18, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 100 \t Average Score (over agents): 95.18 \t Average Score on 100 Episode: 65.67, Time: 00:22:48***\n",
      "Episode: 101, Score: 87.57, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 102, Score: 95.06, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 103, Score: 99.46, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 104, Score: 85.37, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 105, Score: 88.30, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 106, Score: 93.26, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 107, Score: 86.22, Max: 107.98, Min: 45.66, T-max: 1000   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 108, Score: 86.39, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 109, Score: 92.95, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 110, Score: 95.46, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "*** Episode 110 \t Average Score (over agents): 95.46 \t Average Score on 100 Episode: 69.91, Time: 00:25:04***\n",
      "Episode: 111, Score: 96.57, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 112, Score: 107.68, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 113, Score: 103.98, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 114, Score: 105.25, Max: 107.98, Min: 45.66, T-max: 1000   \n",
      "Episode: 115, Score: 125.42, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 116, Score: 98.82, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 117, Score: 94.93, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 118, Score: 91.78, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 119, Score: 102.73, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 120, Score: 97.88, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "*** Episode 120 \t Average Score (over agents): 97.88 \t Average Score on 100 Episode: 74.87, Time: 00:27:10***\n",
      "Episode: 121, Score: 109.84, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 122, Score: 99.40, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 123, Score: 93.42, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 124, Score: 106.46, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 125, Score: 102.88, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 126, Score: 105.21, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 127, Score: 112.26, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 128, Score: 102.49, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 129, Score: 118.29, Max: 125.42, Min: 45.66, T-max: 1000   \n",
      "Episode: 130, Score: 132.80, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "*** Episode 130 \t Average Score (over agents): 132.80 \t Average Score on 100 Episode: 79.61, Time: 00:29:18***\n",
      "Episode: 131, Score: 125.35, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 132, Score: 123.34, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 133, Score: 117.92, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 134, Score: 118.95, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 135, Score: 124.56, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 136, Score: 110.89, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 137, Score: 118.94, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 138, Score: 131.62, Max: 132.80, Min: 45.66, T-max: 1000   \n",
      "Episode: 139, Score: 136.17, Max: 136.17, Min: 45.66, T-max: 1000   \n",
      "Episode: 140, Score: 129.26, Max: 136.17, Min: 45.66, T-max: 1000   \n",
      "*** Episode 140 \t Average Score (over agents): 129.26 \t Average Score on 100 Episode: 86.19, Time: 00:31:30***\n",
      "Episode: 141, Score: 128.87, Max: 136.17, Min: 45.66, T-max: 1000   \n",
      "Episode: 142, Score: 118.09, Max: 136.17, Min: 45.66, T-max: 1000   \n",
      "Episode: 143, Score: 136.13, Max: 136.17, Min: 45.66, T-max: 1000   \n",
      "Episode: 144, Score: 116.34, Max: 136.17, Min: 45.66, T-max: 1000   \n",
      "Episode: 145, Score: 142.87, Max: 142.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 146, Score: 136.71, Max: 142.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 147, Score: 135.22, Max: 142.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 148, Score: 134.39, Max: 142.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 149, Score: 146.28, Max: 146.28, Min: 45.66, T-max: 1000   \n",
      "Episode: 150, Score: 136.92, Max: 146.28, Min: 45.66, T-max: 1000   \n",
      "*** Episode 150 \t Average Score (over agents): 136.92 \t Average Score on 100 Episode: 93.26, Time: 00:33:41***\n",
      "Episode: 151, Score: 146.52, Max: 146.52, Min: 45.66, T-max: 1000   \n",
      "Episode: 152, Score: 153.44, Max: 153.44, Min: 45.66, T-max: 1000   \n",
      "Episode: 153, Score: 145.44, Max: 153.44, Min: 45.66, T-max: 1000   \n",
      "Episode: 154, Score: 155.97, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 155, Score: 146.66, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 156, Score: 141.29, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 157, Score: 151.83, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 158, Score: 145.71, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 159, Score: 134.30, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 160, Score: 145.89, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "*** Episode 160 \t Average Score (over agents): 145.89 \t Average Score on 100 Episode: 101.20, Time: 00:35:57***\n",
      "Episode: 161, Score: 151.97, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 162, Score: 144.06, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 163, Score: 141.81, Max: 155.97, Min: 45.66, T-max: 1000   \n",
      "Episode: 164, Score: 159.43, Max: 159.43, Min: 45.66, T-max: 1000   \n",
      "Episode: 165, Score: 142.53, Max: 159.43, Min: 45.66, T-max: 1000   \n",
      "Episode: 166, Score: 165.84, Max: 165.84, Min: 45.66, T-max: 1000   \n",
      "Episode: 167, Score: 151.11, Max: 165.84, Min: 45.66, T-max: 1000   \n",
      "Episode: 168, Score: 156.89, Max: 165.84, Min: 45.66, T-max: 1000   \n",
      "Episode: 169, Score: 157.89, Max: 165.84, Min: 45.66, T-max: 1000   \n",
      "Episode: 170, Score: 171.14, Max: 171.14, Min: 45.66, T-max: 1000   \n",
      "*** Episode 170 \t Average Score (over agents): 171.14 \t Average Score on 100 Episode: 109.71, Time: 00:38:02***\n",
      "Episode: 171, Score: 167.94, Max: 171.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 172, Score: 153.36, Max: 171.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 173, Score: 163.89, Max: 171.14, Min: 45.66, T-max: 1000   \n",
      "Episode: 174, Score: 178.58, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 175, Score: 166.16, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 176, Score: 147.46, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 177, Score: 175.76, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 178, Score: 171.11, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 179, Score: 159.70, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 180, Score: 165.96, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "*** Episode 180 \t Average Score (over agents): 165.96 \t Average Score on 100 Episode: 118.98, Time: 00:40:15***\n",
      "Episode: 181, Score: 175.61, Max: 178.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 182, Score: 181.50, Max: 181.50, Min: 45.66, T-max: 1000   \n",
      "Episode: 183, Score: 181.38, Max: 181.50, Min: 45.66, T-max: 1000   \n",
      "Episode: 184, Score: 189.66, Max: 189.66, Min: 45.66, T-max: 1000   \n",
      "Episode: 185, Score: 163.22, Max: 189.66, Min: 45.66, T-max: 1000   \n",
      "Episode: 186, Score: 170.11, Max: 189.66, Min: 45.66, T-max: 1000   \n",
      "Episode: 187, Score: 197.68, Max: 197.68, Min: 45.66, T-max: 1000   \n",
      "Episode: 188, Score: 203.09, Max: 203.09, Min: 45.66, T-max: 1000   \n",
      "Episode: 189, Score: 194.66, Max: 203.09, Min: 45.66, T-max: 1000   \n",
      "Episode: 190, Score: 171.05, Max: 203.09, Min: 45.66, T-max: 1000   \n",
      "*** Episode 190 \t Average Score (over agents): 171.05 \t Average Score on 100 Episode: 129.71, Time: 00:42:39***\n",
      "Episode: 191, Score: 204.88, Max: 204.88, Min: 45.66, T-max: 1000   \n",
      "Episode: 192, Score: 365.82, Max: 365.82, Min: 45.66, T-max: 1999   \n",
      "Episode: 193, Score: 207.89, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 194, Score: 210.45, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 195, Score: 187.19, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 196, Score: 196.92, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 197, Score: 192.01, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 198, Score: 191.07, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 199, Score: 217.43, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 200, Score: 218.17, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "*** Episode 200 \t Average Score (over agents): 218.17 \t Average Score on 100 Episode: 142.66, Time: 00:45:31***\n",
      "Episode: 201, Score: 209.03, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 202, Score: 202.94, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 203, Score: 207.35, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 204, Score: 208.74, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 205, Score: 217.97, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 206, Score: 196.01, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 207, Score: 224.49, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 208, Score: 211.50, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 209, Score: 220.33, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 210, Score: 218.93, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "*** Episode 210 \t Average Score (over agents): 218.93 \t Average Score on 100 Episode: 154.74, Time: 00:47:46***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 211, Score: 206.65, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 212, Score: 207.38, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 213, Score: 223.16, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 214, Score: 236.69, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 215, Score: 233.84, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 216, Score: 235.96, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 217, Score: 226.66, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 218, Score: 238.24, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 219, Score: 211.15, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 220, Score: 217.61, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "*** Episode 220 \t Average Score (over agents): 217.61 \t Average Score on 100 Episode: 166.86, Time: 00:49:49***\n",
      "Episode: 221, Score: 231.79, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 222, Score: 239.67, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 223, Score: 240.89, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 224, Score: 248.11, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 225, Score: 261.12, Max: 365.82, Min: 45.66, T-max: 1000   \n",
      "Episode: 226, Score: 512.83, Max: 512.83, Min: 45.66, T-max: 1999   \n",
      "Episode: 227, Score: 243.24, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 228, Score: 272.20, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 229, Score: 257.68, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 230, Score: 278.05, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "*** Episode 230 \t Average Score (over agents): 278.05 \t Average Score on 100 Episode: 183.88, Time: 00:52:05***\n",
      "Episode: 231, Score: 284.37, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 232, Score: 275.33, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 233, Score: 244.79, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 234, Score: 260.68, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 235, Score: 259.12, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 236, Score: 264.47, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 237, Score: 281.34, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 238, Score: 299.17, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 239, Score: 291.71, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 240, Score: 289.92, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "*** Episode 240 \t Average Score (over agents): 289.92 \t Average Score on 100 Episode: 199.02, Time: 00:54:12***\n",
      "Episode: 241, Score: 247.10, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 242, Score: 288.12, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 243, Score: 496.16, Max: 512.83, Min: 45.66, T-max: 1999   \n",
      "Episode: 244, Score: 245.71, Max: 512.83, Min: 45.66, T-max: 1000   \n",
      "Episode: 245, Score: 519.05, Max: 519.05, Min: 45.66, T-max: 1999   \n",
      "Episode: 246, Score: 281.62, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 247, Score: 267.02, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 248, Score: 271.96, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 249, Score: 288.12, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 250, Score: 254.84, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "*** Episode 250 \t Average Score (over agents): 254.84 \t Average Score on 100 Episode: 217.30, Time: 00:56:40***\n",
      "Episode: 251, Score: 278.21, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 252, Score: 271.08, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 253, Score: 267.94, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 254, Score: 280.15, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 255, Score: 260.06, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 256, Score: 313.75, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 257, Score: 284.41, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 258, Score: 279.29, Max: 519.05, Min: 45.66, T-max: 1000   \n",
      "Episode: 259, Score: 599.58, Max: 599.58, Min: 45.66, T-max: 1999   \n",
      "Episode: 260, Score: 284.67, Max: 599.58, Min: 45.66, T-max: 1000   \n",
      "*** Episode 260 \t Average Score (over agents): 284.67 \t Average Score on 100 Episode: 233.82, Time: 00:58:57***\n",
      "Episode: 261, Score: 554.62, Max: 599.58, Min: 45.66, T-max: 1999   \n",
      "Episode: 262, Score: 304.01, Max: 599.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 263, Score: 284.20, Max: 599.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 264, Score: 305.12, Max: 599.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 265, Score: 537.70, Max: 599.58, Min: 45.66, T-max: 1999   \n",
      "Episode: 266, Score: 300.89, Max: 599.58, Min: 45.66, T-max: 1000   \n",
      "Episode: 267, Score: 614.41, Max: 614.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 268, Score: 265.37, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 269, Score: 288.68, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 270, Score: 557.26, Max: 614.41, Min: 45.66, T-max: 1999   \n",
      "*** Episode 270 \t Average Score (over agents): 557.26 \t Average Score on 100 Episode: 258.52, Time: 01:01:53***\n",
      "Episode: 271, Score: 581.81, Max: 614.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 272, Score: 285.30, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 273, Score: 243.16, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 274, Score: 565.17, Max: 614.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 275, Score: 309.74, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 276, Score: 275.60, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 277, Score: 578.01, Max: 614.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 278, Score: 299.50, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 279, Score: 309.69, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 280, Score: 299.01, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "*** Episode 280 \t Average Score (over agents): 299.01 \t Average Score on 100 Episode: 279.49, Time: 01:04:33***\n",
      "Episode: 281, Score: 281.49, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 282, Score: 602.05, Max: 614.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 283, Score: 317.24, Max: 614.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 284, Score: 628.51, Max: 628.51, Min: 45.66, T-max: 1999   \n",
      "Episode: 285, Score: 293.56, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 286, Score: 278.92, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 287, Score: 289.26, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 288, Score: 292.93, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 289, Score: 303.01, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 290, Score: 285.28, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "*** Episode 290 \t Average Score (over agents): 285.28 \t Average Score on 100 Episode: 296.93, Time: 01:07:02***\n",
      "Episode: 291, Score: 288.36, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 292, Score: 271.69, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 293, Score: 290.97, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 294, Score: 304.28, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 295, Score: 286.75, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 296, Score: 290.81, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 297, Score: 321.01, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 298, Score: 296.45, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 299, Score: 302.02, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 300, Score: 331.31, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "*** Episode 300 \t Average Score (over agents): 331.31 \t Average Score on 100 Episode: 304.85, Time: 01:09:05***\n",
      "Episode: 301, Score: 310.42, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 302, Score: 299.09, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 303, Score: 314.84, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 304, Score: 606.47, Max: 628.51, Min: 45.66, T-max: 1999   \n",
      "Episode: 305, Score: 343.87, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 306, Score: 615.90, Max: 628.51, Min: 45.66, T-max: 1999   \n",
      "Episode: 307, Score: 315.54, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 308, Score: 625.97, Max: 628.51, Min: 45.66, T-max: 1999   \n",
      "Episode: 309, Score: 322.68, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 310, Score: 308.76, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "*** Episode 310 \t Average Score (over agents): 308.76 \t Average Score on 100 Episode: 324.31, Time: 01:11:46***\n",
      "Episode: 311, Score: 329.49, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 312, Score: 337.64, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 313, Score: 337.85, Max: 628.51, Min: 45.66, T-max: 1000   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 314, Score: 361.98, Max: 628.51, Min: 45.66, T-max: 1000   \n",
      "Episode: 315, Score: 721.77, Max: 721.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 316, Score: 336.87, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 317, Score: 352.17, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 318, Score: 356.57, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 319, Score: 341.65, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 320, Score: 347.49, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "*** Episode 320 \t Average Score (over agents): 347.49 \t Average Score on 100 Episode: 340.18, Time: 01:14:03***\n",
      "Episode: 321, Score: 381.17, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 322, Score: 674.16, Max: 721.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 323, Score: 363.54, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 324, Score: 352.78, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 325, Score: 369.42, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 326, Score: 341.48, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 327, Score: 364.45, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 328, Score: 353.11, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 329, Score: 385.92, Max: 721.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 330, Score: 723.96, Max: 723.96, Min: 45.66, T-max: 1999   \n",
      "*** Episode 330 \t Average Score (over agents): 723.96 \t Average Score on 100 Episode: 355.42, Time: 01:16:46***\n",
      "Episode: 331, Score: 373.99, Max: 723.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 332, Score: 371.51, Max: 723.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 333, Score: 362.85, Max: 723.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 334, Score: 357.82, Max: 723.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 335, Score: 374.23, Max: 723.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 336, Score: 358.56, Max: 723.96, Min: 45.66, T-max: 1000   \n",
      "Episode: 337, Score: 802.56, Max: 802.56, Min: 45.66, T-max: 1999   \n",
      "Episode: 338, Score: 358.88, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 339, Score: 380.17, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 340, Score: 395.11, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "*** Episode 340 \t Average Score (over agents): 395.11 \t Average Score on 100 Episode: 369.27, Time: 01:19:08***\n",
      "Episode: 341, Score: 371.06, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 342, Score: 392.76, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 343, Score: 402.43, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 344, Score: 374.99, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 345, Score: 421.97, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 346, Score: 792.96, Max: 802.56, Min: 45.66, T-max: 1999   \n",
      "Episode: 347, Score: 739.93, Max: 802.56, Min: 45.66, T-max: 1999   \n",
      "Episode: 348, Score: 400.26, Max: 802.56, Min: 45.66, T-max: 1000   \n",
      "Episode: 349, Score: 853.10, Max: 853.10, Min: 45.66, T-max: 1999   \n",
      "Episode: 350, Score: 844.46, Max: 853.10, Min: 45.66, T-max: 1999   \n",
      "*** Episode 350 \t Average Score (over agents): 844.46 \t Average Score on 100 Episode: 393.61, Time: 01:22:06***\n",
      "Episode: 351, Score: 394.59, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 352, Score: 428.32, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 353, Score: 803.14, Max: 853.10, Min: 45.66, T-max: 1999   \n",
      "Episode: 354, Score: 442.46, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 355, Score: 813.76, Max: 853.10, Min: 45.66, T-max: 1999   \n",
      "Episode: 356, Score: 413.05, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 357, Score: 440.39, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 358, Score: 426.65, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 359, Score: 435.81, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 360, Score: 408.87, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "*** Episode 360 \t Average Score (over agents): 408.87 \t Average Score on 100 Episode: 412.49, Time: 01:24:35***\n",
      "Episode: 361, Score: 432.91, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 362, Score: 425.07, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 363, Score: 429.68, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 364, Score: 438.73, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 365, Score: 423.69, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 366, Score: 432.64, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 367, Score: 460.11, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 368, Score: 448.60, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 369, Score: 411.76, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 370, Score: 840.25, Max: 853.10, Min: 45.66, T-max: 1999   \n",
      "*** Episode 370 \t Average Score (over agents): 840.25 \t Average Score on 100 Episode: 419.80, Time: 01:26:52***\n",
      "Episode: 371, Score: 442.79, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 372, Score: 438.56, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 373, Score: 448.86, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 374, Score: 475.87, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 375, Score: 434.69, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 376, Score: 451.37, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 377, Score: 458.01, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 378, Score: 441.46, Max: 853.10, Min: 45.66, T-max: 1000   \n",
      "Episode: 379, Score: 889.46, Max: 889.46, Min: 45.66, T-max: 1999   \n",
      "Episode: 380, Score: 898.63, Max: 898.63, Min: 45.66, T-max: 1999   \n",
      "*** Episode 380 \t Average Score (over agents): 898.63 \t Average Score on 100 Episode: 436.13, Time: 01:29:18***\n",
      "Episode: 381, Score: 464.63, Max: 898.63, Min: 45.66, T-max: 1000   \n",
      "Episode: 382, Score: 470.39, Max: 898.63, Min: 45.66, T-max: 1000   \n",
      "Episode: 383, Score: 854.28, Max: 898.63, Min: 45.66, T-max: 1999   \n",
      "Episode: 384, Score: 871.41, Max: 898.63, Min: 45.66, T-max: 1999   \n",
      "Episode: 385, Score: 414.61, Max: 898.63, Min: 45.66, T-max: 1000   \n",
      "Episode: 386, Score: 910.62, Max: 910.62, Min: 45.66, T-max: 1999   \n",
      "Episode: 387, Score: 445.40, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 388, Score: 462.56, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 389, Score: 435.20, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 390, Score: 442.32, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "*** Episode 390 \t Average Score (over agents): 442.32 \t Average Score on 100 Episode: 458.12, Time: 01:31:58***\n",
      "Episode: 391, Score: 470.94, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 392, Score: 447.15, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 393, Score: 449.33, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 394, Score: 456.29, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 395, Score: 469.65, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 396, Score: 467.13, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 397, Score: 441.31, Max: 910.62, Min: 45.66, T-max: 1000   \n",
      "Episode: 398, Score: 936.13, Max: 936.13, Min: 45.66, T-max: 1999   \n",
      "Episode: 399, Score: 468.85, Max: 936.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 400, Score: 482.14, Max: 936.13, Min: 45.66, T-max: 1000   \n",
      "*** Episode 400 \t Average Score (over agents): 482.14 \t Average Score on 100 Episode: 479.17, Time: 01:34:18***\n",
      "Episode: 401, Score: 927.88, Max: 936.13, Min: 45.66, T-max: 1999   \n",
      "Episode: 402, Score: 473.94, Max: 936.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 403, Score: 448.12, Max: 936.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 404, Score: 934.24, Max: 936.13, Min: 45.66, T-max: 1999   \n",
      "Episode: 405, Score: 940.13, Max: 940.13, Min: 45.66, T-max: 1999   \n",
      "Episode: 406, Score: 925.06, Max: 940.13, Min: 45.66, T-max: 1999   \n",
      "Episode: 407, Score: 919.30, Max: 940.13, Min: 45.66, T-max: 1999   \n",
      "Episode: 408, Score: 470.64, Max: 940.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 409, Score: 420.46, Max: 940.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 410, Score: 480.00, Max: 940.13, Min: 45.66, T-max: 1000   \n",
      "*** Episode 410 \t Average Score (over agents): 480.00 \t Average Score on 100 Episode: 507.93, Time: 01:37:39***\n",
      "Episode: 411, Score: 496.79, Max: 940.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 412, Score: 492.97, Max: 940.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 413, Score: 476.59, Max: 940.13, Min: 45.66, T-max: 1000   \n",
      "Episode: 414, Score: 947.74, Max: 947.74, Min: 45.66, T-max: 1999   \n",
      "Episode: 415, Score: 987.41, Max: 987.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 416, Score: 440.42, Max: 987.41, Min: 45.66, T-max: 1000   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 417, Score: 481.76, Max: 987.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 418, Score: 467.07, Max: 987.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 419, Score: 501.55, Max: 987.41, Min: 45.66, T-max: 1000   \n",
      "Episode: 420, Score: 496.19, Max: 987.41, Min: 45.66, T-max: 1000   \n",
      "*** Episode 420 \t Average Score (over agents): 496.19 \t Average Score on 100 Episode: 527.58, Time: 01:40:15***\n",
      "Episode: 421, Score: 982.41, Max: 987.41, Min: 45.66, T-max: 1999   \n",
      "Episode: 422, Score: 998.00, Max: 998.00, Min: 45.66, T-max: 1999   \n",
      "Episode: 423, Score: 494.59, Max: 998.00, Min: 45.66, T-max: 1000   \n",
      "Episode: 424, Score: 506.94, Max: 998.00, Min: 45.66, T-max: 1000   \n",
      "Episode: 425, Score: 514.57, Max: 998.00, Min: 45.66, T-max: 1000   \n",
      "Episode: 426, Score: 485.72, Max: 998.00, Min: 45.66, T-max: 1000   \n",
      "Episode: 427, Score: 1055.72, Max: 1055.72, Min: 45.66, T-max: 1999   \n",
      "Episode: 428, Score: 525.59, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 429, Score: 517.97, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 430, Score: 956.56, Max: 1055.72, Min: 45.66, T-max: 1999   \n",
      "*** Episode 430 \t Average Score (over agents): 956.56 \t Average Score on 100 Episode: 554.86, Time: 01:43:23***\n",
      "Episode: 431, Score: 534.23, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 432, Score: 520.05, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 433, Score: 537.12, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 434, Score: 516.69, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 435, Score: 533.51, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 436, Score: 484.10, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 437, Score: 505.07, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 438, Score: 512.75, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 439, Score: 532.26, Max: 1055.72, Min: 45.66, T-max: 1000   \n",
      "Episode: 440, Score: 983.60, Max: 1055.72, Min: 45.66, T-max: 1999   \n",
      "*** Episode 440 \t Average Score (over agents): 983.60 \t Average Score on 100 Episode: 570.10, Time: 01:45:43***\n",
      "Episode: 441, Score: 1067.37, Max: 1067.37, Min: 45.66, T-max: 1999   \n",
      "Episode: 442, Score: 541.69, Max: 1067.37, Min: 45.66, T-max: 1000   \n",
      "Episode: 443, Score: 1074.77, Max: 1074.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 444, Score: 504.74, Max: 1074.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 445, Score: 996.96, Max: 1074.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 446, Score: 1031.91, Max: 1074.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 447, Score: 1056.65, Max: 1074.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 448, Score: 1113.40, Max: 1113.40, Min: 45.66, T-max: 1999   \n",
      "Episode: 449, Score: 1079.28, Max: 1113.40, Min: 45.66, T-max: 1999   \n",
      "Episode: 450, Score: 1068.86, Max: 1113.40, Min: 45.66, T-max: 1999   \n",
      "*** Episode 450 \t Average Score (over agents): 1068.86 \t Average Score on 100 Episode: 609.52, Time: 01:49:44***\n",
      "Episode: 451, Score: 1046.80, Max: 1113.40, Min: 45.66, T-max: 1999   \n",
      "Episode: 452, Score: 535.06, Max: 1113.40, Min: 45.66, T-max: 1000   \n",
      "Episode: 453, Score: 563.16, Max: 1113.40, Min: 45.66, T-max: 1000   \n",
      "Episode: 454, Score: 557.11, Max: 1113.40, Min: 45.66, T-max: 1000   \n",
      "Episode: 455, Score: 504.79, Max: 1113.40, Min: 45.66, T-max: 1000   \n",
      "Episode: 456, Score: 532.65, Max: 1113.40, Min: 45.66, T-max: 1000   \n",
      "Episode: 457, Score: 535.53, Max: 1113.40, Min: 45.66, T-max: 1000   \n",
      "Episode: 458, Score: 1089.69, Max: 1113.40, Min: 45.66, T-max: 1999   \n",
      "Episode: 459, Score: 1072.80, Max: 1113.40, Min: 45.66, T-max: 1999   \n",
      "Episode: 460, Score: 1125.87, Max: 1125.87, Min: 45.66, T-max: 1999   \n",
      "*** Episode 460 \t Average Score (over agents): 1125.87 \t Average Score on 100 Episode: 635.08, Time: 01:52:46***\n",
      "Episode: 461, Score: 559.03, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 462, Score: 567.15, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 463, Score: 522.76, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 464, Score: 499.29, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 465, Score: 1099.18, Max: 1125.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 466, Score: 555.60, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 467, Score: 1108.91, Max: 1125.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 468, Score: 580.50, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 469, Score: 541.42, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 470, Score: 572.34, Max: 1125.87, Min: 45.66, T-max: 1000   \n",
      "*** Episode 470 \t Average Score (over agents): 572.34 \t Average Score on 100 Episode: 653.71, Time: 01:55:15***\n",
      "Episode: 471, Score: 1158.59, Max: 1158.59, Min: 45.66, T-max: 1999   \n",
      "Episode: 472, Score: 592.24, Max: 1158.59, Min: 45.66, T-max: 1000   \n",
      "Episode: 473, Score: 1122.65, Max: 1158.59, Min: 45.66, T-max: 1999   \n",
      "Episode: 474, Score: 574.30, Max: 1158.59, Min: 45.66, T-max: 1000   \n",
      "Episode: 475, Score: 1149.06, Max: 1158.59, Min: 45.66, T-max: 1999   \n",
      "Episode: 476, Score: 600.69, Max: 1158.59, Min: 45.66, T-max: 1000   \n",
      "Episode: 477, Score: 1163.77, Max: 1163.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 478, Score: 571.27, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 479, Score: 1121.20, Max: 1163.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 480, Score: 574.24, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "*** Episode 480 \t Average Score (over agents): 574.24 \t Average Score on 100 Episode: 686.19, Time: 01:58:24***\n",
      "Episode: 481, Score: 585.58, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 482, Score: 1141.43, Max: 1163.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 483, Score: 566.41, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 484, Score: 564.55, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 485, Score: 599.55, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 486, Score: 1081.23, Max: 1163.77, Min: 45.66, T-max: 1999   \n",
      "Episode: 487, Score: 607.20, Max: 1163.77, Min: 45.66, T-max: 1000   \n",
      "Episode: 488, Score: 1189.23, Max: 1189.23, Min: 45.66, T-max: 1999   \n",
      "Episode: 489, Score: 1149.55, Max: 1189.23, Min: 45.66, T-max: 1999   \n",
      "Episode: 490, Score: 1090.40, Max: 1189.23, Min: 45.66, T-max: 1999   \n",
      "*** Episode 490 \t Average Score (over agents): 1090.40 \t Average Score on 100 Episode: 714.23, Time: 02:01:36***\n",
      "Episode: 491, Score: 571.54, Max: 1189.23, Min: 45.66, T-max: 1000   \n",
      "Episode: 492, Score: 608.06, Max: 1189.23, Min: 45.66, T-max: 1000   \n",
      "Episode: 493, Score: 570.98, Max: 1189.23, Min: 45.66, T-max: 1000   \n",
      "Episode: 494, Score: 1196.87, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 495, Score: 557.74, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 496, Score: 594.90, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 497, Score: 1136.21, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 498, Score: 1136.56, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 499, Score: 618.71, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 500, Score: 1184.71, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "*** Episode 500 \t Average Score (over agents): 1184.71 \t Average Score on 100 Episode: 745.10, Time: 02:04:29***\n",
      "Episode: 501, Score: 586.75, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 502, Score: 557.24, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 503, Score: 1070.01, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 504, Score: 553.20, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 505, Score: 1155.20, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 506, Score: 601.44, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 507, Score: 557.03, Max: 1196.87, Min: 45.66, T-max: 1000   \n",
      "Episode: 508, Score: 1193.86, Max: 1196.87, Min: 45.66, T-max: 1999   \n",
      "Episode: 509, Score: 1225.06, Max: 1225.06, Min: 45.66, T-max: 1999   \n",
      "Episode: 510, Score: 615.54, Max: 1225.06, Min: 45.66, T-max: 1000   \n",
      "*** Episode 510 \t Average Score (over agents): 615.54 \t Average Score on 100 Episode: 756.86, Time: 02:07:26***\n",
      "Episode: 511, Score: 557.27, Max: 1225.06, Min: 45.66, T-max: 1000   \n",
      "Episode: 512, Score: 579.10, Max: 1225.06, Min: 45.66, T-max: 1000   \n",
      "Episode: 513, Score: 565.95, Max: 1225.06, Min: 45.66, T-max: 1000   \n",
      "Episode: 514, Score: 634.08, Max: 1225.06, Min: 45.66, T-max: 1000   \n",
      "Episode: 515, Score: 1245.49, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 516, Score: 1179.22, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 517, Score: 540.43, Max: 1245.49, Min: 45.66, T-max: 1000   \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 518, Score: 1203.06, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 519, Score: 1207.55, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 520, Score: 620.01, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "*** Episode 520 \t Average Score (over agents): 620.01 \t Average Score on 100 Episode: 782.30, Time: 02:10:22***\n",
      "Episode: 521, Score: 599.54, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 522, Score: 1189.25, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 523, Score: 576.92, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 524, Score: 591.79, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 525, Score: 570.08, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 526, Score: 620.79, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 527, Score: 628.96, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 528, Score: 594.24, Max: 1245.49, Min: 45.66, T-max: 1000   \n",
      "Episode: 529, Score: 1215.60, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 530, Score: 1228.80, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "*** Episode 530 \t Average Score (over agents): 1228.80 \t Average Score on 100 Episode: 790.08, Time: 02:13:12***\n",
      "Episode: 531, Score: 1189.87, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "Episode: 532, Score: 1196.51, Max: 1245.49, Min: 45.66, T-max: 1999   \n",
      "\n",
      " End of PPO!\n"
     ]
    }
   ],
   "source": [
    "def ppo(env, agent, n_episodes=2000, print_every=10, n_agent = 12):\n",
    "    \"\"\"\n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        print_every (int): number of episodes to print result\n",
    "        n_agent (int): number of identical agents in environment\n",
    "    \"\"\"\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    scores_global = []    \n",
    "    scores = []\n",
    "    max_t = 2000\n",
    "        \n",
    "    time_start = time.time()\n",
    "    \n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        states = env_info.vector_observations\n",
    "        agent_scores = np.zeros(n_agent)\n",
    "        t_max = 0\n",
    "        for t in range(max_t):\n",
    "            actions, log_probs, _, values = agent.act(states)\n",
    "            # get needed information from environment\n",
    "            env_info = env.step(actions)[brain_name]\n",
    "            next_states = env_info.vector_observations\n",
    "            rewards = env_info.rewards\n",
    "            dones = np.array([1 if t else 0 for t in env_info.local_done])\n",
    "            agent.save_step([states, values.detach(), actions, log_probs.detach(), rewards, 1 - dones])\n",
    "            states = next_states\n",
    "            agent_scores += rewards\n",
    "            t_max = t\n",
    "            if all(dones) or t == max_t-1:\n",
    "                agent.step(next_states)\n",
    "                break\n",
    "                \n",
    "        score = np.mean(agent_scores)\n",
    "        scores_deque.append(score)\n",
    "        scores.append(score)\n",
    "\n",
    "        print('Episode: {}, Score: {:.2f}, Max: {:.2f}, Min: {:.2f}, T-max: {}   '\\\n",
    "              .format(i_episode, score, np.max(scores), np.min(scores), t_max))\n",
    "        \n",
    "        if i_episode % print_every == 0:\n",
    "            torch.save(agent.actor_critic.actor.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.actor_critic.critic.state_dict(), 'checkpoint_critic.pth')\n",
    "            s = (int)(time.time() - time_start) \n",
    "            print('*** Episode {} \\t Average Score (over agents): {:.2f} \\t Average Score on 100 Episode: {:.2f}, Time: {:02}:{:02}:{:02}***'\\\n",
    "                  .format(i_episode, score, np.mean(scores_deque), s//3600, s%3600//60, s%60))\n",
    "            \n",
    "        if len(scores_deque) == 100 and np.mean(scores_deque) >= 800: # 650: # 500:   \n",
    "            break\n",
    "\n",
    "\n",
    "    print('\\n End of PPO!')\n",
    "    return scores\n",
    "\n",
    "scores = ppo(env=env, agent=agent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEKCAYAAAAFJbKyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztnXmYHVWZuN+v9ySdTmcl+x62sCWEfSfsMOI6CIwiMiKKP1B0FHAQl3EZl0FRZERBQJQdhJEIhBAEZE1ICFnIRkL2pLN1J+m97/n9Ucutqlu3btXt7ttLvvd5+rlVp06dOtXpnO9+6xFjDIqiKIoSl6KunoCiKIrSs1DBoSiKoiRCBYeiKIqSCBUciqIoSiJUcCiKoiiJUMGhKIqiJEIFh6IoipIIFRyKoihKIlRwKIqiKIko6eoJdAZDhgwx48eP7+ppKIqi9Cjmz5+/3RgzNFe/Xik4xo8fz7x587p6GoqiKD0KEfkwTj81VSmKoiiJUMGhKIqiJEIFh6IoipIIFRyKoihKIlRwKIqiKIlQwaEoiqIkQgWHoiiKkggVHIqiKN2c7XubeHbx5q6ehosKDkVRlG7OF/80n2seeIdd+5q7eiqACg5FUZRuz9a6RgD2NLZ28UwsVHAoiqJ0Qxpb2mhLGQD6lVnVofY1Zxccv3lxJc8t2VKQuangUBRF6YYcfMuzfOmB+QD0LS8GoD6L4Njb1MrPn1/BF/80vyBz6zTBISL3iMg2EVnsafuZiLwvIotE5EkRqfZcu0lEVonIchE519N+nt22SkRu7Kz5KoqidBda2lIAPL90K+DROJraQvu/tWZHYSZm05kax73AeYG22cBhxpgjgBXATQAicijwaWCqfc9vRaRYRIqBO4DzgUOBS+2+iqIovRbHp+HQtyytcazZvo+UbcJy+KBmHwBVFYUpeN5pgsMY8zKwM9D2vDHG0bXeAEbbxxcDDxljmowxa4BVwLH2zypjzAfGmGbgIbuvoihKr2VLrSU4hldVANCv3BIIt89ZxRk/f4mXVmzz9Te2HKkrkPO8K30cnwf+bh+PAtZ7rm2w27K1K4qi9Fi21TUyZ9nWrNc32YLjgAGW4HA0jqWb64DM6CpDWgN5ZtFmjPFrJB1NlwgOEfk20Ar82WkK6WYi2sPGvFpE5onIvJqamo6ZqKIoSgdRs6eJxhbLR/Gvv3udq+6bl3WB3+oIjv7lQFrjcEgZ4zNneS1Xtz69mM6m4IJDRK4ALgIuN+nf2gZgjKfbaGBTRHsGxpi7jDEzjDEzhg7NufOhoihKQTnmhy9w+R/eBGDtjnogbWJ6dvEWdtenk/scAVNRamkapcX+79C3/HUJx/1oDnsaW3zjAPzoY4cjEvadu+MoqOAQkfOAbwEfMcbUey49DXxaRMpFZAIwBXgLeBuYIiITRKQMy4H+dCHnrCiK0lHM/3CX79wA2/Y0cs0D832htEE9JKiY7G2yTFUNzW12f6vD298+i3OmDu/QOYfRaS54EXkQOB0YIiIbgFuxoqjKgdm2RHzDGHONMWaJiDwCLMUyYV1rjGmzx/kK8BxQDNxjjFnSWXNWFEUpJMYYmlut0NsNuxrc9lRAUuTyWDjdq/oUJqqq055ijLk0pPnuiP4/BH4Y0j4LmNWBU1MURekWpLJIhKCGERQkmf2t60WdbKJy0MxxRVGUdtDY0uZmdC/eWMv4G59h4frdse41WXQJE/jMpXI4AqgwYkMFh6IoSrs47WdzOfQ7zwEwZ9k2+zN7qG2bR80IKhKtbSlum73CdXo7mkZujcP67GynuENhDGKKoii9lK11TYAVbnvXy6tz9t/pKY1ujF94PLVwE7+aszLdYNL9oki5pqp4c24vqnEoiqJ0ADc8spB9zelaUs2tKRas25XRzxt2GzRVNds1qhzSGkf0s53LhdI4VHAoiqJ0AMFyHz+atYyP/fY1Vm3bk7Xfa6t2cNsLK7KO6Wga2Xwh3o4FkhmAmqoURVE6hSWbagHYua/F1+74LwC+9Of5tLRlFwqOxpHNVGXcfoVzjINqHIqiKLH41mOLeOOD7OXLt9Q2ZL3mLS3irTMVJTTAE12VRXJ4NZJCheKCCg5FUZScNLW28fC89Vz6+zey9nGc5AC/fnEVb69N+ze8636S7V8dgZFNvHh9IIU0VangUBRFsfn9yx+EhtLWNljmpf7l+Vn3vQu/11QVJLj2OwInWziu8fQrlGMc1MehKIri8sNZywBY+5MLfe11tuCo6lPqto2/8Rm+fvaBscZNZTFV5cIrGELHTTk+EKM+DkVRlO5EbYO12PevsATHa6u3A/CL2dkjorz4TVXZNY4gScJx1VSlKIpSYJxig2G4GkdFCcu37OGy37+ZaOy8NY4cNUdcwZJS57iiKErB8WoC3rIgjS1tXHnv24ClcezyJPDlQ5LtXdOCIfx6OqpKw3EVRVEKjndB37EvHSG1dsc+97h/RX5u4ZeWp/cIb80mBSLIlgDozfMopMahznFFUfZr1u+sJ2WMa44CWLejnmH9rf2+y0uK3fZ8BMeGXfXc8Mi77nmUvyK49ufycRhvP/VxKIqidD6NLW2c8tO5nPazl6jzmKpmL02H5Hr9E8V5VBHc1+Q3TWVL5tvb1MrvXv4g0Nf/GcTbrj4ORVGUAnDwLc+6x3V25NSQynJeXrndbfcuznl9sQ8s6NmEQG1DCx/U7PO1uaaobPt2eMqua1SVoihKB7Bjb5O7L3cuHOf4cRMH8UHNXl+OhEM2bSEJufbW8JJT4/D0U+e4oihKB3D0f73Ax377z1h9d9j7ZBw9diBNrSk27rZqT3n9C7nyKUIJ7h+eYIy04MgRjms0HFdRFKXDeH/LntD2VEAKrNtRz5DKco4YPQCAVdv2An4zUc7y5jFIpHGQwznuDcdVU5WiKErn0tDiN2Gt3bGP8YP7MriyHEjXp/JGz+alcQRIonGkPIIh/Ho6HLeQtapUcCiKsl8SFBwf7qhn/JB+7var6W/77fRxBBb0ZD6O6D3HvaYs9XEoiqJ0MkGn+Za6RsYO6uv6CsLy9Drim30S0ZOj4ojPea4+DkVRlE6mPiTaanhVhXucCvm2n0RbyEaSMRxTVfay6r0sHFdE7hGRbSKy2NM2SERmi8hK+3Og3S4icruIrBKRRSIy3XPPFXb/lSJyRWfNV1GU/YugqQpgWFU5RbatKiwUNi+5EbgpkZ8kx9axXh9Ib9E47gXOC7TdCMwxxkwB5tjnAOcDU+yfq4E7wRI0wK3AccCxwK2OsFEURWkP9c2ZxQaH9a+I9HHk4xwP3pLET5JT48jhA+ksOk1wGGNeBnYGmi8G7rOP7wM+6mm/31i8AVSLyAjgXGC2MWanMWYXMJtMYaQoipKYsMTAYVXlaR+Hu2inr+cTjtsWkDaJ8jiI3jrW6wPpFaaqLBxgjNkMYH8Os9tHAes9/TbYbdnaFUVRQnn47XU8tXBjzn5BU1VJkTCob5kbnZT+Fu+Nqko+n6DgSOTjSDnPza1x7I/VccPeOFuJ+dDfoIhcjWXmYuzYsR03M0VRegzGGL71+HsAXHxU9HfMoHN8QJ9SiorEjZoyIRpHPiahTMER/95cW8fuLwmAW20TFPanU6R+AzDG0280sCmiPQNjzF3GmBnGmBlDhw7t8IkritI9+df/fZ2v/OUdALdMSByCpirHKe76OEIc03lpHBklR5LncWRPAEzPq7c4x8N4GnAio64AnvK0f9aOrjoeqLVNWc8B54jIQNspfo7dpiiKAsBba3fyt0WbAVi8sQ6AvmXFUbcAVkl1L47AkAwfR/vCcYOlTfKpVRXHOV7IBMBOM1WJyIPA6cAQEdmAFR31E+AREbkKWAd8yu4+C7gAWAXUA1cCGGN2isgPgLftft83xgQd7oqiKAA0tVrCoF957qUtaDIS/BpHWB6HIXn2eGt7fBwJwnELaarqNMFhjLk0y6WZIX0NcG2Wce4B7unAqSmK0ksIag35LMoOQY0jxDeOMcnjqoIaR7Iih9H3uFFXxmitKkVRlDg4pdAdnCikOEtoUHNwFt5MjcM/flJzVVDjSHJ3rmd5kxS1VpWiKEoMduxt8p2nPM5kr2CY9v3nWbKpNtDXP1aRvRoGNY6MsuoJVQ6vc7y4SJI52HP6ONLXe7NzXFEUpcPYbguOshJrKcsWAbWrvoW7X13juzfTVBVD4zDJs8fb2jyCQ6SDfRzp6705HFdRFCUvWttS1OzxaxjOnhlVFaWAXzvItUBnOsctHAES5l8wJnn2eHs0jtw+jvSn+jgURdnvWb+znmcXb3bPv3D/PI754Qtu5BSkfRppLcH6FHJblII+DkdgSEDjCDrHk2ocXud4cVEyjSPXnuNpjUP341AUReGiX7/KNQ+8Q0NzG9v3NjF3eQ0AizemfRXBRThJzkXwsiMwnLDcsByK9objJtU4cpmq8AiWogKu5t2l5IiiKIoPxwx1yHee9dnv31yzk6PHDQIyF1RfQcIcC3ROH0fK7+MoEuuepDmAKdMBGkcW/cm3H0cBdQ7VOBRF6fZ419p7Xl3r5m8EF2FviY6M9TlCyIBXcPh9HM6Y1qKf3MfR6nGOF0lCH0eIg96LWwSRtMArBCo4FEXpMppbUz7TUy6OHjeQ7Xub2FrXCITtdZE+zm2qCuZx+D+DUVXWom9Ct5SNwu8czy8BMGt1XJy5UtCwKhUciqJ0Gd/9vyVc9OtXWb+zPlb/smJryQrWkQou9nGc48EFXFznuL9WlTOS459oT+a4FY6b4N6cRQ7VOa4oyn7A6pq9rsawcN1uIO3PyEVJsbOoh5twkpRAzzRV+Y+DZqLiIsFgEmeOeyxVdgXe5D6ObMLGG3WlpipFUXolza0pZv7iH1z6+zfyur/Y3Q/c+D4dvOf5Osed42BUVXGRkEolL63e5rFtlRQl1TjsgxzVcQ1aq0pRlF7KP1ZYIbVbaxvzur+kKFDyPOWYpvyaCOQOmw1e9n5jF8kMhS0WS+NIHI7rdY4njKoCv9YTflU1DkVRejEbd1m+jKmjBuR1f1rjsM6jTVXRY2XzcTjHQT9KkRtVlQxfOK5IRrXc6Hutz2yRXN7S7xqOqyhKr8Sb2Z0PxUVBH0fQVOU9zuXjCI+qAr+Pw322E1XV3gTABPe6fpYskVy+6riqcSiK0hvxLrqPzFvP0s11ie4vLnKiqsIzqv2Z47nm4j/P8HEExnSjqhL7OAKCIw8fR+6oKhUciqL0UryL5jcfW5T4/pKAqSpowvE5x4PXMuYSdI6njwVP5rj9bd/J+k6qcbQvczw8CCCjH2qqUhSll5LPnt1eMk1VwfGtT5EYUVUB848ENI7gt/2kZiYHr8ZRlLCseo6gKp+vp5C1qlRwKIoSiycXbOCGhxe2awxnDd0e2IApLhlRVVkSAOOYlLJtHeuMF/SjWLWq2meqKklaVj3wnkH8CYCqcSiK0s34j0cX8cSCjYlKhARxFrr65rYcPcMJahzZihwacms3mftxeDQOrxTxJACSh6nKp3EkFhxOnka26+kpqo9DUZRux8lThgDw2urtsfq/vnoHd8xd5WszgW/x6fZ4cwgmAKbzOPwDxSkNkq1WlTNeUOMoLipqt8aRfAfA8Lmmr6dNdoVMANSy6oqixKJvWTHgDy+NwskOv/aMyW5b2sSU3xyKM0xV/uvehTZXvkR0OK4Q9KM4BQqTlxxpRzgu4ZpV+rpzYDQBUFGU7kt7/Ntem7yXuF+WXR9Hyq8NZIwfY55RpirxOcdtjUPyK3LY5nHCJ42qyhWO662npUUOFUXptiTJfA7irJltgTHim6r81XGDPgCv0zwYjhtcWKMW8CJPVJZbVt1e9JOWHPHWqkqax4HnfUIve4RbUW+vVSUiXxORJSKyWEQeFJEKEZkgIm+KyEoReVhEyuy+5fb5Kvv6+K6Ys6IoFs5C+sYHOzIEQC6cRTdo7or7Lbwk4ONICwy/JpNK5d4bPOqR4s0cd3wczgZP7fBxJF3cc20dm67Z1cud4yIyCrgOmGGMOQwoBj4N/DdwmzFmCrALuMq+5SpglzFmMnCb3U9RlC4iZQxvfLCDT9/1Br95cVXuG3z3Wp/ewn/OmHHI9HH4/RDOKFY4bkCryZhLlMaR6eMochMAY03VxZ85nuzeoGDMvO4VoL1c48ByyvcRkRKgL7AZOBN4zL5+H/BR+/hi+xz7+kwpZPiAouxHbKlt5JlFm0OvOWtXbUMLNz/5HgCravYmGj/lahypQHu8+7MlALpmpVR6IU1e5DB9XBRS5DDt48jfOV6SMEsvrsZhCuwcL3hUlTFmo4j8HFgHNADPA/OB3caYVrvbBmCUfTwKWG/f2yoitcBgIF5MoKIosbnkrtf5cEc950w9n9LA12NnEbv3tbV5j+9qHBk+jqQah985Htx0yWrPFVUV/ayApYqS4vw0Dq+MLEq4uhuPYIjq0OtrVYnIQCwtYgIwEugHnB/S1flNhf06Mn6LInK1iMwTkXk1NTUdNV1F2a/YuKsB8H/DfWfdLsbf+Axb8txDw4vJ8g067mKcUasqYKLyCpTcgiHCVFWUmXPibu7UjgTA4jwWd2Oy6ziuaW4/cI6fBawxxtQYY1qAJ4ATgWrbdAUwGthkH28AxgDY1wcAO4ODGmPuMsbMMMbMGDp0aGe/g6L0arxL1QNvfAjAuxvyyxj3LtC5SmfkIkPjCITlejWPpJnjXrw+DmcYJwcjqcbh1a6SahzO87P+3lLp9+3VGgeWiep4Eelr+ypmAkuBucAn7T5XAE/Zx0/b59jXXzRJ4+EURUmE939YVUVp1n5x1irvWNkW3aRRVcEEQO8Caj3TJK5V5aUoJI/DESZJlx/vc0ryERzkTgA0ppdvHWuMeRPLyf0O8J49h7uAbwE3iMgqLB/G3fYtdwOD7fYbgBsLPWdF2d/wLnaV5e1zhaZiaByx8ziK/ftxeBP+rM+0ltAejUPwmr/sZxc548abq0Nrmz+PIylRZjdvcEAhI4a6pOSIMeZW4NZA8wfAsSF9G4FPFWJeiqJYeBeqyor2Co70cfbooJimKjeXwgQ+/c9KxdA4ojSH0D3HnQTAWDNN0548jvTzswnctODs1RqHoijdn+6qcQRNVUGnuNfHERwzKCii9hwvsreJ9fYrLioKzQ/JhT8cNz+NI0dQFakCh+Oq4FAUJQPjCSEtzScUyEMcwZG3czyocXjmnWvMqMtuBBVpQVEs8XwnQdoTjuuOkSsBsMCmKhUciqJk4F2o2lLZ+8WxjnhNVdls9bE1juIszvGAIAH/N/3weSUzVVmZ4/GFXNg8ivMwJ0WZx7wO/N4ejqsoSjcnjpaQz1i59pXIRXA/jmw+DsgspJj5zOzXxJc5bj9bhMzSibnxlxzJMxw3y2R9mpaaqhRF6Ur8WkI7BYdnsOA+32HPi8L5xp7O47Da0zWbvJpSLlNVwMfhOS4Sr+PZ8XHkp3H45p+vjyPimoNqHIqidCm+pL12lFGHeEIosY8j5b8vWLPK6pO/xuH1cXiLHJoQp3sS8tI4iLO3iFEfh6IoHcO2usbIBXTdjnoem78hoz2OXyIufrNXeJ+4kUppH0dQYHSmj8MgYud25JEA6CWvcNxUhInPKepoVONQFKUD2FzbwLE/msMv56zM2uejv/0n33j03chQ1cgFNsY84vk4YgxEeiOnYMRT2Fayuba4zdgB0LvnuDdz3F6Ui0RiffuPIj+NI3sCoDccuVuWHBGRk0XkSvt4qIhM6LxpKYrSXrbVNQHw0vJtWfvs3NcMhBUdjCc44qyhcUJkk5cccfwPnrkY/37gQU0r+ITIIofi1WosM5DT1h4NLD8fR/ZS7q4ApRvWqhKRW7FKgtxkN5UCD3TWpBRFaT9J1regWcd7GhWOG4c4pqrk+3Fkjm0MvpfO3GUw+7yCeNdgg6VxiAipVD5xVWnyyxyPKjmSNlV1x8zxjwEfAfYBGGM2Af07a1KKohSW4CLaWaaq7JnjCZ3jAROV0xalcWS8Y4RA9O8AaPs4JN4GUVHklzlO1m8BTrPpps7xZrsirQEQkX6dNyVFUQpNcBH115dqn3fcmPDjbH2iCOZxBLWZKOd4zpIjnuMikXSor8F2jue3H4eXJJnj7rticgpcRysqFHEFxyMi8jusPTO+ALwA/L7zpqUoSiGJ0jiiM8dzL1YdWXIko1aV11QVcCIH8ziCwjHqkcGoKss5Hm02ikOS6i2ujDHZzY7pkOHCOsdjVS8zxvxcRM4G6oCDgO8YY2Z36swURSkYUd/OI53jMRZ87wKeXXDkHAYI28jJOxf/fDIERw6Nw4slONJzs3wc0Y7qJPOPg6VBGLtgY7TALXQ4bk7BISLFwHPGmLMAFRaK0oOZ/+FOpo4cQEVpsa/dRJiq2l9yJPzY3yeuxuHsx5F5nzHRJUeSOMeLRGizpVI6qkoiQ2Pj4IQTx+vrNVWF9/GG4xaSnG9hjGkD6kVkQAHmoyhKB+N8D12/s55P3Pk6335ycUafoMYR2zke41uuz5zUQc7xYP6GdWwiNaXgM6IeWRTI47Cc40JYufYkFCfInEuXV8nex31HU9hw3LiF9huB90RkNnZkFYAx5rpOmZWiKO0muFDuaWwFYMmmzL3Do/wB7Q/HDT/O1ieKoKnKa/0P5lhkhuPmcI57Vt7MzHHLVFXIzPEit7xKxPM8mle3MlXZPGP/KIrSC4mKOGpvVFU8H0d+znFf9FdgnMxw3OAzsz9HvPtxYDmqLed4svyYICUJvOOOkIwq1uhNhOx2W8caY+4TkTLgQLtpuTGmpfOmpShKRxPl1I1KAOzIsuodrXH4BFzKP++kGocXb3Vc59u8YOV2tKfoYyKNw+4b/LcR8ZZcSX/mu0lUPsQSHCJyOnAfsBZLsI0RkSuMMS933tQURWkP7nITK2Q2eB4vHDfWPLxaQTt9HCWuj8M6j0oAzBVVFdfHkbL9B0V2AmB7xGiSqCrHHxIUVOKZg9c53u00DuAXwDnGmOUAInIg8CBwdGdNTFGU9pHkm3FUlnUhMsfjzrU4YPf353FEPyszyTFzQXYo8vk4bP+Hbb5q134cHaBxBLPawalV1f0SAEsdoQFgjFmBVa9KUZRuSra1OGyByTTjZL+WFO/Cl63MR9wnRIXjWlFV6b7tMVWBf89xp8ghJBPIQZLncWRqTt5/Pt8cu1uRQ2CeiNwtIqfbP78H5nfmxBRlf+aGRxZy32trE983d/k2GprbgMwFJ2qdzOwbT+OIQ5yx4q7FThpEWAJgUHBkVMfN4Rz3LrxBjcMpqw659/mIIpmpyurb2OKXtl7Bb4zhf/+xmpY2QwFdHLEFx5eAJcB1wPXAUuCazpqUouzvPPHORm59ekmie5Zv2cOVf3yb//yrlaeRzW8Qtr5ERRy1fyOn9HG2NTeuj6PIExZrje0Pq4ryzeTK6wg+J+1HsRZl5/fWnt9HPrWq9ja1+tq9IzS1pvjJ398H2pdfkpS4gqME+JUx5uPGmI8BtwPFOe7JiohUi8hjIvK+iCwTkRNEZJCIzBaRlfbnQLuviMjtIrJKRBaJyPR8n6soXc2qbXsi98fIl137mrns928AsGb7XiA6qS9IZHXc9m4d24HhuM43/9CtYoOCI8f7R4fjeiO3rG/5sfIqcpDMx2F97rXzb8psb7k363/ltr3u8a76wgW6xhUcc4A+nvM+WIUO8+VXwLPGmIOBI4FlwI3AHGPMFPt5N9p9zwem2D9XA3e247mK0qWc9T8v87k/vt2hY7alDD+atYwd9qZMDs76Jp5+UWP4741pqoqxDrbFGCvuWuxEN6Wdwl7neLDIoV/lSFpyJB255PcfRP0ecxEWjvvkl0/k0mPHZLQ7AmL7XmtDrn7l1vnEoeni5Lvq0//mNXua8p5XUuIKjgpjjCva7OO++TxQRKqAU4G77bGajTG7gYuxQn6xPz9qH18M3G8s3sCq0Dsin2crSk9g9tKtifr/4vnlPOrZN9yxgbuRR3Z7Eo3D+BbgRNPJIKgVxHl+NtwNlULDcf1O9pzO8QgBIAEfhyWw2u/jCFM4+leUMKSyPKO9qsKKP9pc2whAv3IrCHby0Eq3z26PllGzt/sJjn1eE5GIzAAa8nzmRKAG+KOILBCRP9j7exxgjNkMYH8Os/uPAtZ77t9gtylKt2b5lj2Mv/EZ5iYwTTW1tvGF++cles6L74ePH1woowRAVKhqpP8hxhoaZ6y4a3E6gzvTx5FKBWpVJc7jSK/qXh+Ht6w65KdxTBjSj/d/cF6o4BBbGAYZ0McSHFtqraW20hYcBw1P76Hn1Ti2d0ON46vAoyLyioi8DDwEfCXPZ5YA04E7jTHTsGpf3RjRP0wZzvwnF7laROaJyLyampo8p6YoHce8D3cC8PyS+BpERzo4k5iq8g3HjaMpxKpVFXMxFtfHkfY/ZJtPhnM8h7Pc/xy/j8NxylvjJv9HGlJZRkVpcaipqqRIQhc5R3AENY5B/cp45IsnAP6/l6PHDUw8r3yJFBwicoyIDDfGvA0cDDwMtALPAmvyfOYGYIMx5k37/DEsQbLVMUHZn9s8/b0GwNHApuCgxpi7jDEzjDEzhg4dmufUFKVw1DVmOjPzCX3NlviVK3Pa1zfCkdwWMaU4a2g853juccDROLzOcf/YwfBc/zMCGkfkc/xRVU5Z9bBxAE4/KHrNGda/Agj/Fjy4sjxcoNh1rYKCA+DYCYN8fV/+jzP46SePiJxDR5JL4/gd4OhCJwA3A3cAu4C78nmgMWYLsF5EDrKbZmKF9z4NXGG3XQE8ZR8/DXzWjq46Hqh1TFqK0lN5dN56jvju86yyo2JeWr6NlVv3tMvxGiRoFooaO7PkeDzneBx7f7xaVfE1Dr824B87ZYxblqQ1oGJk5nHE9HFATuf4F0+dFDnvYVWWD6PezrHx0q+smMGVZRntLbbKtG5nPQCV5dkDWccO7puxx0pnkqvkSLExZqd9fAlwlzHmceBxEVnYjuf+P+DPduHED4ArsYTYIyJyFbAO+JTddxZwAbAKqLf7KkqP5qG3Lbfd9r1NTB5W6UZavfudcxKPFfyu6iz6wUU9apGPMutEmZHi5F/E2b88ibj0aQO+HQBNuthfyuTM44hOAAxkjnsTAEN8Rc4qcy7+AAAgAElEQVS95SVFNLWmO5QVF9HclnI1ju0hDmwRYVR1n4x2pwy+d6zuQq6ZFIuII1xmAi96rsWtc5WBMWahbVY6whjzUWPMLmPMDmPMTGPMFPtzp93XGGOuNcZMMsYcboxJ5jlUlC4nc1l0vkXWNbSwaMNut709ETtBMuz/dkOYZSsyjyPKxxEj4irOWElKtxdFaBwG4+ZKZIbjZteqgghWcp2xzV9Fkv69vbAs02fVp7SY2y+dxnNfPZXjJ1pmpJ9+8gjX3HSArXFk80OMDBEcXsd3GCv+6/zI651JrsX/QeAfIrIdK4rqFQARmQxk7gajKEosnJj7q//kr9zTkVuABjWFSOd4xN4VsXagi8AEFvd8x3HwOscDTyJl0hV0g5pBrpIj/pEs7eCWpxZjsKKq1u/MHkg6orqCI8dUA/DQ1Se47fPX7uLheeuZMsyKhJo4tJK1P7mQlrYUU779d7ffyOqKjDG9obYzDx6Wcb2spIgFt5zdrmq9+RKpcRhjfgh8HbgXONmk/wKKsMxNiqLk4MG31rNm+77cHckvKzlbMrKzuDrXo6Oj/OdRYa1R94XRFuGwTjKOgz+PI9PHUVwc7sROUuTQEewPvLHO9UtERS0N6ZeZhwHw3588ggW3nM3ho/07b5faZicnma9/RWbN2DGD0qlyd3/umNAgiIH9yhjUL9M/0tnkNDfZSXfBthWdMx1F6Z1ccc9bvPzNM3L260hTVVDDiMrjyIyq8h63Nxw33Sdb92Qah3/Pcdul4RY5TDvHs/t4HH9INjbtTmsXNXuaKBLhwiNG8Mx7w5n13hYAfvDRw7jFrgsWVYNqYJaF/ZVvnkF137TA+Pv1p3D5H95kzMA+3HTBIRw6sord+7rnfnl5+ykURYlPU2tmNE0Y+URVZdM4ggtjlFCKSpaLDsdNKjiy+ThyDuNSJOL6VqwoKssBbWzh4YbNRuyjHvY876/RKzh21TcztL+lUQy1M7wnDunHZ44f5wqOfPBqFACHjKjinVvO9rVVhWgi3YHu46ZXlF6MMXDzk+/F6tdRZGgRiRIAcy/2Vr/c84hVciSBwAyWPPduJ+s9DwrhjbsbuOHhhWypbXTvP+fQA0Kfsc8TNru1rgknRc/RHpzf7YxxA/n0MZl1pjqDM2w/x6EjqwryvChUcChKJxHMB/7Lm+ty3tOReRyujyPG2M41p4d/sW9fOK73uSkTvuFQe3wcjqBwNA5XcITM7YkFGzn+x3NYurkOgCPHVPO//zbdHjfd74unTvTdN8n2RTj+hNoGy4T02JdO5CefKEzi3UeOHMnS75/LwcNVcChKr8XkEe/SET4OZ4RgkcNIU1VGxJHHVJWgqm742H4fR0mIPyDRXhlF/lpVznDG+MePmtubH1jpaaMH9gnV8m664BBev+lM9/z6sw4EYGBfS3DsLmAJcy99y7qHd0EFh6Lkyc59zXzyztfYXJu73mdccdDevS98Y7UrjyP7OFH3OZiAsPD2D9sFLzNzPfszfbWqUn5TVcoY11Hd3Jo9GuCddbsAf8HAIE6tKIAxA608C8fXsb+jgkNRPDS3prj6/nms2LonZ9/H529g3oe7+MMrazrs+floHOEl8tLfuF1TVYLoqLgbOWXPy8gylu3MzjVOrr0yvLWqiu3xjDu+9cavrtoOwGdPGMfNFxzM7ZdOc8d4bfUOSouFiUMqswYX9PGU8Cixw2fHD+4X3nk/o3voPYrSTVi0YTfPL93Kjn3NPP6lE9s1VrYFPYo4mdixx0rgHM93z/FsJqVsO/GlTPi+20GzXpT4DNaRKvFpHJmbJX3rvIPdAoFnHTKMQ7/zHLUNLZwyZQhlJUVZtZuwvAknA3x/RwWHonhI8n0/wS6gsenIzPGMcNzImlPBeYQfB4mTCR6sVRXu48h+fxCnjlRtfQttKeMm0z23eAs1e5p8moLT38F77bJjxwK4Gd+XhERHTRzaj6NGV7vn2SoR72+o4FCUPHj63U381zPLgHghtHHlQUfmcWQWOYz/3Li1qrLN1wSERXqscOd4Mh+H5V868vvPAzDWzod4zN4FMVhI0Pv78S78h42ysrlHVvdh7U8uDH3Wi18/PaPtM8ePY19Ta2bn/QgVHIqSB39+40P3OFv0VFdFVTlLYyJTVZ6Z43FMVcH9OEJ9HAk2WSoSYcH6Xe65s6+J4xR/5rpTuOfVNfzJ/jfKJljDKtLG4QcfPSyv+3oT6hxXlBA6wiCRjwxIUiXWIdtcM4ocJhAA3vPILWdjOMe9mk4qla4l5e+fXXAFEREaW9KTckJja/Y0ccLEwUwY0o8ffPQwLjvOMkUVZ5EcUWVClGhUcCiKh7jrdpxu+fgrohbpuLh5HPaBY56Jzsfwn3uFTnTmeG6NY+e+Jm5+8j3qm1sxBkpjRFVF5nHY6/2QykxH9cB+6RDaH1x8GIu/d64bEeXwwg2n8vzXTs06vpIbNVUpSj74bPjhXfLxV3Rk5ribDe7mPDjhucm+8eeTAOgVPHfMXQ3A4aMGROZxPLVwI4/N38Adl0/nu08tyfpMx9k9amAfrj1jEi1tKX40630gXdUWrOityvLMJW7ysOy5G0o8VONQlDyI47/IRwjkFVWVxRTjCAxnxLzzOCJNXOHtDS2ZRR2Li8StYgtw6Ih06YwnFmzk+ocW8srK7Tz01jqeWLDRvXZ1oPyHExlVVVHClSdN4Kgx6XLnx00YnHWuSsehgkNRPDiLbUdEXfoX3HgCoUM1Drcshz2flHOe+YygP8QfFZX9GWFjvb56Byf8+MWM9qcXbqKhpY3jJw7mS6dP4t4rjwkd01vT69Z/OZSbzj/Yd32AXYq8f4WlTTgKTEVpEV89a0r2ySodhpqqFMVD3HU7jmKQirn4+u/pQOd4QGA4giRMOAVDdbMl8GU+I/PaW2t2hvZ1MrlTxkrKy1Zqfu2Oeve4Zk9TRu5EtV0KpH+59ensuzF15IAMf4bSOehvWVE8xF24fbpElnsKZqrKNlZAw2jz7GERJPgO8Tdyymyrb47OcViyydp1Oiqz/sunTwJg/a7MOmDVAY3DiapyChAqnY9qHIrioTXCgZyUfAoWdkRUlfv8oKkqSuOISgCMmNOa7ft48f2tnH7gMETgvtfW8vLK7ZHz6ltm+SjComFnHjyM62ZOQQR++9Jq+pRa323funmmu0dGpa1plJVY1xzNJWzfbqVzUMGhKB7iLva+6q9Z+rTFNPf47unIzHFHw0j5BUbYIyL3HA+Z+xkHDWXj7gZWbN3L5++dx5DKcj5x9Ch+948PfP2OmzAIY+CttZb56oGrjnMr0gZLgTS0tPGLfz2SaltzuPPy6Zw0ZQgAw6rSQsGxRjlzvuDwEayu2Zexh4bSeajgUBQPwX2qsxEvjyN9HFcgdISpygQ0jKCmkd49L7twCJqqRPx+mqH9y1lVs9c93763KUNoAJwzdThXnTyBt9fupL65jZNtQQB+gffEl0+ktqHFFRoA5x8+IvT9ijxFDQFKi4u44ewDQ/sqnYMKDkXxEHeBjxN15NVeOlNwZDOqmSyCI0zziDJVtaWs7OtWT9uAPqVs2t2YdU4XHTGCvy3azA67btQx4wdlztsjOQ4ZEX9XOycTvCP3LlGS0WXOcREpFpEFIvI3+3yCiLwpIitF5GERKbPby+3zVfb18V01Z6X34y6gHRCO6zVPtUZVGAx7fgKCdwTzNpwh3fOAALH6BJIFPYPua2p1/RIOw/pXuPd/bNooAL5+9oEs/t65PHz18Zw4ydIsduxtTvw+ubjoyJEM7lfGpXZJEaXwdGVU1fXAMs/5fwO3GWOmALuAq+z2q4BdxpjJwG12P0XpFFpjbojhz9DInT0dd9wkW6iGPcd7T9ApHgzLDStE6N0gCaCxpY2GljZ3r22HYVXlnDv1ACBd7XZQZRmV5SUcN3Gw68cYPTC6kOCMcQP55SVH5XxHL6Oq+zD/lrOZNLQy0X1Kx9ElpioRGQ1cCPwQuEEsnfVM4DK7y33Ad4E7gYvtY4DHgN+IiJh8qsEpSg5im4pi9PN/o483bEbNKAMhNQH99wTLp6f8giLTVIWv3XlO0OdhjOFnzy0HsHwPnvyKoZXl3Hn50bSkUtTsaWL73iYuOmKke/3ocQN58ssncrhdujwbj7Vzsyyla+gqjeOXwDcB57/JYGC3McYJAN8AjLKPRwHrAezrtXZ/Relw4pqUvGT1ceTx1Saq9Ec2gsIm6AR3lJ2gpuEVbG327nkOK7fu5dF5G7j71TUAHDM+XdYDLI2jqEgoLylm9MC+/PHKY317dANMGztQE/J6KQX/VxWRi4Btxpj53uaQribGNe+4V4vIPBGZV1NT0wEzVfZHnEU1l4sjXlRVHgmAwVLoMaRP8J5sCX9RYbnWtqvphueXbuWbjy9yz884aBivfPMMN+R1xID89rJQegddYao6CfiIiFwAVABVWBpItYiU2FrFaGCT3X8DMAbYICIlwAAgo6aBMeYu4C6AGTNmqBlLyYvY4bjeqKosffJxdAefH0f4BP0nQT+FK0gcn0fAlOUcRz2rum8ZYwb15ZvnHcy1Z0529/BW9k8KrnEYY24yxow2xowHPg28aIy5HJgLfNLudgXwlH38tH2Off1F9W8onUXsBED8i24Y+ezm19waLgSiyNivO8MJbs/HFjBO+676dMST5ePI/gxnn4viIqGqojR7R2W/oDsZIL+F5ShfheXDuNtuvxsYbLffANzYRfNT9gPckiMJwnGzaSn55Bm0BBwWcbSWYJ9sVXG372l2+++ub2ZLbToP4+21O11HeBhaB0rx0qX6pjHmJeAl+/gD4NiQPo3Apwo6MaVHsWxzHR/u2Md5h4VnGichnwTArJsZ5aFxBAVHHOGTITiyRFVt3WMJiqbWFEd9fzan2FncHz1qJH9duIlFG2ozxv7dZ45mUL8yKkqLM64p+y9qqFR6POf/6hUA1v7kwnaP1ZGCI5+ChS2BqK44wicjEivECd6WMj4NA+AVuxjh9z96GHOX11Db0OK73hG/T6V30p1MVYrS5eRTq6ozNY44fpJspqrNdZagqG1oYdLNs2hqTXHWIQdk3F9VUco3zj3I1/bxaaMy+imKg2ociuIhal/ubGTLCs8nqirDOR5DawkKKEf2rN2+L6Pvvxw5gheWbXXPj7VrSH3m+HGcOmUIIwb0ccuVK0o29C9EUTw4GkecPcUdOtXHkYfGkTKGXfuaM0xPAFNHVrHmxxe4549cc4J7PG5wPxUaSixU41B6DcaYjG1GkxLc/CjqWQ4dKTia24LaQzzBcdAB/Vm+dY97fvWf5vn6XHj4CMYP6cvkYVYNqX+dMVojpZS8UcGh9BraUoaSXIWdctAaiEjKhlcmZPOL5Fr0i4sko09rAo3jjrmr2FLbSMrAyVOGuIKjtqGFt9fu4oSJg/n6OQfy5pqdXHvGZN+9P/3kkZFzU5QoVHAovYbWlKGknVGjweS5bLTF0DhyRVWFCY5MU1X4HHfWN7t5F31KiykO2Yf1N5dNY3BlOTNC9sJQlPaggkPpNTS3pdqdb9AaCGXNhvd61gTAHMKnOMSs1hwjAfD2F1fyyxdWuucNLW2+bVgBKstLGFxZHvl8RckXFRxKr6GlNY/EiQDBXfKyPsuzwIeZtU748Rw212bfIQ/S+1h4Wbu93nceJnxmvbc5o62sWBhSWcZ2e+OkvU2tGX0UpaPQEAql1xA3ByOKuIIjSuMwxuQUGpDeO9vL0s11VJQW8dvLpwOW4NhS20hzawpjDM8t2RJqvjpoeBWvfPNM5v3nWQCcduDQnM9XlHxRjUPpNQRzIPLBdY6HfNNvaG7jdy+v5vMnT/AJi6CQaYo5jzC/BEBjS8o1Pe3c18x5v3yFzxw/jgsOH8EX/zQ/9J5pY6vpU1ZMn7JiXrjhNA6oUjOV0nmo4FB6DR2hcThmp7CxnluyhV++sJIVW/f4op+CgiMsf8LhM8eP409vfAhkCo7+FSXsaWz1Xfu/d63dBZ5dsiVUGPzk44fzwrJtjKxO748xeZhuqap0LmqqUnoNwYikfAiG4+7aly497oS7znpvC7vq08KhLWX48axlvLR8GwB1EYLj+rOmuMdBH8fL/3GGezyk0sqxePCt9QDU7Gni58+vcK9/5MiRvHXzTD597Fj+cMWMBG+oKO1HBYfSa+gIU1V60yPDs4s3M+0Hs3ltlVUMcPHGWsYO6ptxz459Tfzu5Q/43B/fpqm1jUfnbwgde8ygPr5Iqj5l/ggwr89j6sj0Xt1fPG2ir9+0sdV8/+KpDKuqSPh2itIxqKlK6TV0hKnKGaOuoZXbZlshrw+8+SEnTBrMextrOW/qcNbt9Ec+OZFMAAf957O+a0ePG8gdl01nd0MzQyvLfcIhuEc3wB2XTefdDbspKyli5IAKNtU2cuN5B7Nq617mvL+NG88/mGtOm9Tu91SU9qCCQ+k1dISpyjFR1Ta0uL6Kd9fXsmFXA7vrWzhs1AB4e32ssZ657mQmDa2korSY4QMs7WBPY9qMVW0Ljs+eMI6WNkP/8hIuPGIEFx5h7Svy96+e6pZROW7iIOa8v41xIRqPohQaFRxKr6FjfBz+MYZXVbBxdwMPvb0OgCNGDwi7LZThVRUZCYklRZZ1+JARVW6tqMNGDuBfjxmTcb9XI/n8SRMYN7gf5xyaWRZdUQqN+jiUXkNwE6R82F3vd2yfbO+Sd8fc1UwZVsmhI6oy7nnr5pmcdcgwwCrzcdwEq8RHVYgpqk9ZMXdfMYMHv3AcA/pa1+saszvTHUqKizh36vB2F3FUlI5ABYfSo/FWqW1v5nhDcxsL1u12z48aU83Znm/4d312BiXFmf9lhlVVcNpBluA4eHgV93zuGGZddwqlIX0BZh5yANV9y6juY2kcQWGlKN0dNVUpPRqvlpFtQ6W4zF2+zVcr6r4rj2X9LssRPnJABROG9ANgwpB+rAlskvRvx43lvKnDGdrfyrU4dGSmZhKk2tY4dtU35+ipKN0LFRxKj8br1wjuZRHF2u37qCgtZtW2vbQZw8C+pfzHo+8yfnBfTpw8hL+8uY6qPiVMLq3ktAOH8vVzDnTvffjq41m2ZQ/b9zTRv8L6LyQirtCIy/ETBwNwim0OU5SeggoOpUfT6tU4YjrH9zW1cvrPX/K1nTJlCCLCby6bztSRVXz/I1MRESpKi7nv88f6+g6rquiQHIqDhvdn+X+dR3l7a8ErSoFRH4fSo/GalqKiqlbX7KWxpQ2Ad9btyrj+ysrtfP6k8Rw2agAiEurL6AxUaCg9ERUcSo/G69dwTFWtbSkWb6x1BcXa7fuY+Yt/cPAtz/KXN9fx9pqdFAnMuu4U31gnT9GKsooSh4ILDhEZIyJzRWSZiCwRkevt9kEiMltEVtqfA+12EZHbRWSViCwSkemFnrPSfWlpzTRV/fz5FVz061f51ZyVzP9wp88sdfOT73H7i6s4dGQVh46s4o9XHuNeO3JM/BwNRdmf6QqNoxX4ujHmEOB44FoRORS4EZhjjJkCzLHPAc4Hptg/VwN3Fn7KSnelxaNxfO//ltLcmmLZ5joA7n9tLd99emnofTPGWbkWEwZbkVKnTBmiZiNFiUnBnePGmM3AZvt4j4gsA0YBFwOn293uA14CvmW332+sgP03RKRaREbY4yj7KcYYvv+3pVSW+/+E1+3cx/qd9VT3LWV3fQvvbax1r/3msmn837ubeG7JVmbaCXvjh/Tj3iuPcSOcFEXJTZdGVYnIeGAa8CZwgCMMjDGbRWSY3W0U4C0OtMFu8wkOEbkaSyNh7NixnTpvpev5++It/PGfa93zb19wCD+ctYwLfvUqzW0prjltEmUlRUwc0o+vPrwQgHOnDuecQ4fT2NpGVUU6q/v0g4YFh1cUJYIuExwiUgk8DnzVGFMXUUoh7EJGwL4x5i7gLoAZM2a0v/aE0i1pbk1x/+trWV2z19d+3ETL9OREWY0d1JfLjrO+QDS1tvGPFTVuJndZicaEKEp76BLBISKlWELjz8aYJ+zmrY4JSkRGANvs9g2AtwLcaGBT4WardCdu+etiHp6XWZ12yrD+7vHHp43ijIPTEVKXHDOWS45RLVRROoqCCw6xVIu7gWXGmP/xXHoauAL4if35lKf9KyLyEHAcUKv+jf2D22av4OhxAxk+oIIpwyq577W1PqFx0uTB/HPVDgB7r+1Tqe5bxpBK3W9bUTqTrtA4TgI+A7wnIgvttpuxBMYjInIVsA74lH1tFnABsAqoB64s7HSVQtKWMjy5YCPHjh/Er+asdNs/MX00c97fyoEHVLJiq2WmmjS00hUcAJM9WoeiKJ1HV0RVvUq43wJgZkh/A1zbqZNSupwVW/fw8ooaxg7qyzcefTfj+uPvWNux/vKSo/jKXxawt6mViUP6cfpBQ909LhRFKQxaq0rpchas28XHfvsaADPGDYzse/LkIUwbW80rK7fzsemj+dxJEwoxRUVRPKjgUArCss11/OL55by7oZb7rjyWvmXFjKiuYOG63Vxy1xtuv3kf7mJQvzJ27vOXGj9x0mD+/ZQJlBQX8ZtLp1PX2BK6Z7eiKJ2PCg6lU0mlDC8s28o1D8zH3s6bC25/BYBzpx7Ac0u2+voP7FvKnBtO4/P3vc3XzjqQz97zFgB/+cLxbp8BfUvd3fMURSk8KjiUTuG5JVt4euEmTpkyhBufeC9LH7/QGNq/nLnfOJ3K8hKe/PJJAPz7yRNCt2BVFKXrUMGh5MW2PY0MrSx398Bubk3xo1nLOP+w4Wzd08R1Dy4A4LXV29175n7jdOoaWrjxiffcelJfPn0S722sZUCfUr529oEZJUT+86JDC/RGiqLERQWHkph1O+o59Wdz+eZ5B/Hl0ycD8IdXP+De19Zy72trfX0N1lar182c7G69+vfrT+GOuav49Ysr+drZB2bdm1tRlO6JCg4lg68+tICUgdsvnRZ6/Y01Vu7ET59dzvCqCh5/ZwNravZl9DvjoKHc87ljCCsnc+0Zk7n2jMkdO3FFUQqCCg4lg78utCq6ZBMc89daO+gdUFXODY+kcy6+dtaBbKlr4MG3rOzuhpa2UKGhKErPRm0EHUAqZbDyFHs+dY0t7nFrW4rWthQvLN3qvt8P/raUh+et58yDh3Hnvx3NWYekK8teeMRwvn7OQZxxkFUn6qqTJxZ28oqiFATVODqAiTfP4vzDhnPnvx3d1VOJZE9jC5XlJVm1gKcWbuT6hxa65+t3NfDHf67h/tc/5ILDh7Ng3W421zYCMH1sNdPHDuQPVxzDL19Ywda6Rrfkxx+vPLbzX0ZRlC5DBUcH8ffFW7p6CpHUNbZwxHef52tnHcj1Z03h2cVbWLtjH9ecNgmwNkb67dzVvntufHwRb67ZCcCs9/zvN21sOsP7q2cd2MmzVxSlO6GCo500trR19RSy0tTaxo69zQzqV8aCdbsBuO2FFdz2wgq3T3NrisNHD+APr3zA8q17uG7mFK48cTyn/WyuKzQczjpkGL+5bDqrtu3lsFG6P7ei7K+o4GgndQ0tuTt1ET+e9X5GeGyQ/5mdFiI3nX8wnz95AqXFRfzpquO46r55fOOcAzln6nC+9fgibjz/YCpKi1VoKMp+jgqOdrK7CwTHltpGykuKGNivjLXb9/HKqu2MHdSX+qZWzp06nKIiYcfeplChcd7U4Rw5phqD4fcvf0B9cxtFInx8+ii+aJutAI4cU81bN8+kqMjyh/z+szMK9XqKonRzVHC0k9pOFBzGGH40axlnHnwA/1hRw5rtezlp8hC+89QSjhpTzR2XT+f0n7/ku6dPaTGfmjGax+dbZcivOW0Slxwzhl+/uJJLZozhuImD3b7/fvLEyG1UHaGhKIriRXpLGKmXGTNmmHnz5hXkWbOXbuUL91vPWvnD8xNlQe/c18zrq3dw+KgBvP7Bdh6fv5GSYkEEqvuUUV5SxBMLNuY1r6PHDeT7F09l6kg1KymKEg8RmW+MyWleUI2jnXg1jrqGFgZXllOzp4mm1ja++Kf5fPO8g+lTWswRowewfW8TTy3cxLa6RjbubuSFZf4if2MG9QFg/c6G0Gf9+8kTeGPNDg4c1p8nFmxkaP9ybv2XQ1m/s4GqPiVUlpfw/JKtPPPeZm44+0AVGoqidAoqOBKwY28T9c1tLN+yh+q+pfzlzXU+c85P/v4+Bvjbok00tqQAuMIuCz52UF/2NrVm7DNx5UnjWb+zga+cOZkjRw+guS3Fii172dvUyoA+pVT1KeHk/57L+MF93YJ/d728GhZYJT0uOmKkb7xzpw7nI0eN5MRJg1EURekMVHAEeHLBBo6bMJj3t9Sxets+Fq7fzcjqCiYNreTbf11MWyrTtDd6YB+GV1Xw6PwNDKksY1DfMjbZiXIORQKThvbj4auPZ0CfUh6Zt57PnTQhoxpseUkxh4/2awov/8cZ9Ckrds/PP2wEd7+6hi+ckpmZXVFazLlTh7fnV6AoihKJ+jg8OFVfvVSUFrnaw7Sx1Rw2cgBHjammqMha5Fdt28snjh7NqOo+tLSlXB/Hog27KRJh2eY6Tpg0mNED+7b/xRRFUToR9XHkwZhBfZj9tVN5ZN56Dh1ZxZkHHUD/ihIenree1pThkhljIqOQvI7xI0ZXA2jOg6IovQ7VOBRFURQgvsah1XEVRVGURKjgUBRFURLRYwSHiJwnIstFZJWI3NjV81EURdlf6RGCQ0SKgTuA84FDgUtF5NCunZWiKMr+SY8QHMCxwCpjzAfGmGbgIeDiLp6ToijKfklPERyjgPWe8w12m4uIXC0i80RkXk1NTUEnpyiKsj/RUwRHWJlWXxyxMeYuY8wMY8yMoUOHFmhaiqIo+x89RXBsAMZ4zkcDm7poLoqiKPs1PSIBUERKgBXATGAj8DZwmTFmSZb+NcCHeT5uCLA9z3t7CvvDO8L+8Z76jr2H7vCe44wxOU02PaLkiDGmVT7ZG/YAAAZnSURBVES+AjwHFAP3ZBMadv+8bVUiMi9O5mRPZn94R9g/3lPfsffQk96zRwgOAGPMLGBWV89DURRlf6en+DgURVGUboIKjkzu6uoJFID94R1h/3hPfcfeQ495zx7hHFcURVG6D6pxKIqiKIlQweGhtxRSFJF7RGSbiCz2tA0SkdkistL+HGi3i4jcbr/zIhGZ3nUzj4+IjBGRuSKyTESWiMj1dnuveU8RqRCRt0TkXfsdv2e3TxCRN+13fFhEyuz2cvt8lX19fFfOPwkiUiwiC0Tkb/Z5b3zHtSLynogsFJF5dluP/HtVwWHTywop3gucF2i7EZhjjJkCzLHPwXrfKfbP1cCdBZpje2kFvm6MOQQ4HrjW/vfqTe/ZBJxpjDkSOAo4T0SOB/4buM1+x13AVXb/q4BdxpjJwG12v57C9cAyz3lvfEeAM4wxR3nCbnvm36sxRn8sP88JwHOe85uAm7p6Xu14n/HAYs/5cmCEfTwCWG4f/w64NKxfT/oBngLO7q3vCfQF3gGOw0oSK7Hb3b9brDynE+zjErufdPXcY7zbaKxF80zgb1glhnrVO9rzXQsMCbT1yL9X1TjS5Cyk2MM5wBizGcD+HGa39/j3ts0V04A36WXvaZtwFgLbgNnAamC3MabV7uJ9D/cd7eu1wODCzjgvfgl8E0jZ54Ppfe8IVn2950Vkvohcbbf1yL/XHpMAWAByFlLspfTo9xaRSuBx4KvGmDqRsNexuoa0dfv3NMa0AUeJSDXwJHBIWDf7s8e9o4hcBGwzxswXkdOd5pCuPfYdPZxkjNkkIsOA2SLyfkTfbv2eqnGk6e2FFLeKyAgA+3Ob3d5j31tESrGExp+NMU/Yzb3uPQGMMbuBl7D8OdV2/Tbwv4f7jvb1AcDOws40MScBHxGRtVj77JyJpYH0pncEwBizyf7chvUl4Fh66N+rCo40bwNT7GiOMuDTwNNdPKeO5GngCvv4CiyfgNP+WTuK43ig1lGduzNiqRZ3A8uMMf/judRr3lNEhtqaBiLSBzgLy4E8F/ik3S34js67fxJ40dgG8u6KMeYmY8xoY8x4rP9zLxpjLqcXvSOAiPQTkf7OMXAOsJie+vfa1U6W7vQDXIBVhXc18O2unk873uNBYDPQgvXN5SosO/AcYKX9OcjuK1jRZKuB94AZXT3/mO94MpbqvghYaP9c0JveEzgCWGC/42LgO3b7ROAtYBXwKFBut1fY56vs6xO7+h0Svu/pwN964zva7/Ou/bPEWV966t+rZo4riqIoiVBTlaIoipIIFRyKoihKIlRwKIqiKIlQwaEoiqIkQgWHoiiKkggVHIoSQETa7Aqmzk9kpWQRuUZEPtsBz10rIkPyuO9cEfmuiAwUEd1eWel0tOSIomTSYIw5Km5nY8z/duZkYnAKVsLcqcA/u3guyn6ACg5FiYldFuNh4Ay76TJjzCoR+S6w1xjzcxG5DrgGq+z7UmPMp0VkEHAPVhJYPXC1MWaRiAzGStYcipXMJp5n/RtwHVCGVbzxy8aqW+WdzyVYVZwnAhcDBwB1InKcMeYjnfE7UBRQU5WihNEnYKq6xHOtzhhzLPAbrJpKQW4EphljjsASIADfAxbYbTcD99vttwKvGmOmYZWYGAsgIocAl2AVxTsKaAMuDz7IGPMwMB2rfP7hWNnl01RoKJ2NahyKkkmUqepBz+dtIdcXAX8Wkb8Cf7XbTgY+AWCMeVFEBovIACzT0sft9mdEZJfdfyZwNPC2Xe23D+nid0GmYJWlAOhrjNkT4/0UpV2o4FCUZJgsxw4XYgmEjwC3iMhUoktkh40hwH3GmJuiJmJvPzoEKBGRpcAIe++O/2eMeSX6NRQlf9RUpSjJuMTz+br3gogUAWOMMXOxNiaqBiqBl7FNTfaeE9uNMXWB9vOBgfZQc4BP2vs2OPtSjwtOxFjbjz6D5d/4KVbhvKNUaCidjWocipJJH/ubu8OzxhgnJLdcRN7E+tJ1aeC+YuAB2wwlWHtm77ad538UkUVYznGnjPb3gAdF5B3gH8A6AGPMUhH5T6zd4oqwqhxfC3wYMtfpWE70LwP/E3JdUTocrY6rKDGxo6pmGGO2d/VcFKUrUVOVoiiKkgjVOBRFUZREqMahKIqiJEIFh6IoipIIFRyKoihKIlRwKIqiKIlQwaEoiqIkQgWHoiiKkoj/DwfABybAJUNvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1425e710>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
